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Many nonprofits and government agencies are looking for ways to support the 
needs of low-income families by helping parents take steps toward economic 
security while improving their children’s well-being. Programs that offer 
coordinated services to parents and their children—sometimes called 
two-generation programs—provide many different combinations of supports, often 
by partnering with other organizations (Sama-Miller and Baumgartner 2017). 
Participating in research and evaluation can help program leaders and staff refine 
their services and program models and find better ways to coordinate supports for 
families. 


Stakeholders in these programs may find research and evaluation to be useful in a 
number of ways. For example, program leaders and staff may want to serve families 
better but, without planning for and using research and evaluation, may have trouble 
fully executing their vision for high quality, coordinated services that meet the 
needs of parents and their children. And program funders can use research and 
evaluation results to learn whether the programs they support are fully implemented 
and are ready to participate in an evaluation to assess whether they are having an 
impact. Impact evaluations could be premature for most programs serving parents 
and their children because the models for most such programs are not yet fully 
developed. On the other hand, many kinds of research and evaluation are 
appropriate for emerging programs. Moreover, using research and evaluation 
findings can strengthen program implementation and provide valuable insights for 
funders and program leaders about the quality and reach of the program’s services. 


Engaging in research and evaluation can: 


© Help program staff articulate their goals for parents, children, and families and 
see whether their operations are in line with those goals. 

e Show program leaders the quality and intensity of the services they actually 
provide versus the levels of quality and intensity they intend to provide. 


This brief describes 
a continuum of 
research and 
evaluation that 
could help program 
leaders and staff 
create more robust 
programs offering 
coordinated services 
to low-income 
parents and their 
children. 
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e Inform stronger program implementation and demonstrate a 
program’s readiness for an impact evaluation. 


This brief describes a continuum of research and evaluation 
approaches that could help program leaders and staff create more 
robust programs offering coordinated services to low-income 
parents and their children. The brief can also help program 
funders plan descriptive research and evaluation that can inform 
program development and identify which programs are ready for 
an assessment of impacts. 


The brief first discusses how to build a foundation for data- 
informed program improvement, including how to use program 
data to assess whether the program is working as its leaders and 
staff envision. Then, it discusses how research partners can be 
helpful in supporting program development based on analysis of 
program data. As programs develop data systems and program 
services, a descriptive study can examine the types, intensity, and 
quality of program services across several programs. This will 
yield a picture of similarities and differences across program 
models that aim to support low-income parents and their children. 
Programs that have engaged in data-informed program 
improvement may be ready to participate in an impact evaluation, 
so in the final section, we present criteria to help program leaders 
and funders assess the readiness of a program for an impact 
evaluation. 


BUILDING A FOUNDATION FOR 
DATA-| NFORMED PROGRAM 
IMPROVEMENT 


For programs that have not already done so, the best place to 
begin thinking about a vision for program services and how those 
services might influence outcomes for parents and children is with 
a logic model. A logic model is a diagram that helps program 
leaders (1) articulate their plans for services, including the 
intensity (including duration) and quality of services, (2) ensure 
that the plans line up with the expected outcomes for parents and 
children, and (3) identify expected outcomes and the associated 
measures for the outcomes. 


After developing a logic model, program leaders can use it to help 
identify data needs. To ensure that research and evaluation results 
are relevant to and actionable for the program—without involving 
burdensome data collection—program leaders and staff should be 
involved early in planning the broad goals. Over time, leaders and 
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staff should provide ongoing input into the questions to be addressed and the data to be collected. 


DEVELOP A LOGIC MODEL 


Creating a logic model generally involves five steps (Exhibit 1). A program’s leader and staff could start by 
adapting the conceptual framework for services and outcomes developed for this project (see Sommer et al. 2017). 
He or she could also adapt the framework from “Making Tomorrow Better Together,” the report that summarizes 
the activities of the Ascend Network’s two-generation outcomes working group (Ascend 2016). Either framework 
could be adapted to a specific program’s context and goals. 


EXHIBIT 1. STEPS TO CREATING A LOGIC MODEL 


Describe the target population: the families’, parents’, and children’s characteristics 


Articulate the services to be provided to parents and children, including the intensity, quality, and duration 
of services 


Describe the expected outcomes for parents, children, and families in the short and long term, considering 
the intensity, quality, and duration of services for parents and children 


Identify how to measure the intended and actual intensity, quality, and duration of services 


Identify outcome measures for parents, children, and families 


Source: W.K. Kellogg Foundation (2004). 


The logic model summarizes the expectations of program leaders and staff for the services provided to parents 
and children — their intensity, quality, and duration — and the outcomes expected when parents receive those 
services. 


Once the logic model is in place, the program’s leader and staff should consider how to measure services and 
outcomes to assess how well the actual program experience matches the logic model. They can identify measures 
of services and outcomes from large-scale studies of programs for children and families, such as the Head Start 
and Early Head Start Family and Child Experiences Study, and from measures compendia.' We describe possible 
measures later in this brief. 


CREATE OR ENHANCE A PROGRAM DATA SYSTEM 


Once services, outcomes, and measures are defined, research partners, program leaders and staff can work 
together to create a plan for using existing program administrative data to measure services and shorter-term 
outcomes identified in the logic model. They can then consider whether and how to fill gaps in the measurement 
plan by expanding program data collection. 


Administrative data. Program leaders may already have administrative data that meet their needs. However, in 
many cases, program administrative systems cannot easily combine data on parents and their children, 
longitudinal tracking is unreliable, and data items that measure some components of the logic model (such as 
employment or education services received from partners or the quality of early childhood education) could be 
missing or incomplete. 


But program leaders can still draw on the data that are available to assess program services received and shorter- 
term outcomes for parents and children, using the logic model to benchmark whether the program services and 
client experiences are lining up with expectations. Program leaders and researchers can make plans early for how 
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to collect data that will support more complete monitoring and analysis of program activities and outcomes to 
more fully cover the services and outcomes in the logic model. They can work together to decide how to prioritize 
and when to make changes to the data system. They may find that their plans and decisions point toward waiting 
until funds and staff time are available to address the data system needs. 


To provide information about how program services and client experiences line up with expectations summarized 
in the logic model, a program’s administrative data should capture information on participants, participant 
feedback, and the quality of services. 


e Participant information. Staff should gather participant data throughout the participants’ time in the 
program, including (1) characteristics of families using the services, as captured at enrollment; (2) types of 
services received and attendance over time in employment training programs, education programs, and early 
childhood education programs; and (3) shorter-term outcomes for parents and children, such as the 
completion of education and employment programs, children’s language, early literacy, and social-emotional 
development, and other outcomes from parent surveys.” 


e Participant feedback. Staff can gauge participant satisfaction based on feedback and exit surveys to 
understand their level of engagement and reasons for leaving the program. 


© Quality of services. Staff can capture service quality by examining information on staff characteristics and 
training (for example, whether staff in child care centers have certain types of credentials in early childhood 
education) and through other measures, such as observational assessments of early care and education 
classrooms. 


External data. Research partners can provide access to useful external data sources. For example, data on 
outcomes after the program ends could come from administrative data systems, such as state unemployment 
insurance systems (for parents’ employment and earnings) and state or local education data systems (for 
children’s school achievement and progress). Parent surveys and child assessments could also provide outcomes 
data. 


Once these foundational pieces are in place, program staff and research partners can use them to ask and answer 
questions about how program services and outcomes reflect the expectations summarized in the logic model. 


COMPARE SERVICES TO THE LOGIC MODEL AND ADJ UST AS NEEDED 


Program leaders and researchers can use the logic model in two ways. 


First, they can compare their data with the logic model to assess whether the program appears to be working as 
planned. Program leaders and researchers should assess the actual intensity, quality, and duration of services to 
parents and children and compare what they learn with the expectations summarized in the logic model. They can 
then fine-tune their service delivery or enhance service quality and assess the operations again. Exhibit 2 shows 
some examples of how to do this. As they work to improve the program, staff may also find ways to enhance the 
data system itself, which will strengthen their ability to understand and improve program operations. 
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Exhibit 2. Practical examples of how stakeholders can compare data to the logic model 
and consider how to adjust program operations 


Ask a question based on the logic model. Program leaders may ask whether parents are remaining in 
education programs long enough to obtain a credential or a degree, as the logic model suggests. 


Look at administrative data to answer the question. Administrative data can show parents’ 
participation over time. If the data show that many parents are not staying in education programs long 
enough to obtain a credential or a degree, program leaders can then examine the data to find key points 
at which parents leave the program (for example, after a semester). 


If the data do not agree with the logic model, ask staff or parents why program experiences 
are not as expected. Staff can talk to parents, or they might already know, reasons for leaving 
education programs before obtaining the credential or degree. 


Consider strategies to bring families’ program experiences in line with the logic model. Program 
leaders and staff can use the information on when and why parents leave education programs to 
brainstorm ways to re-engage these parents. Researchers can help by suggesting evidence-based 
strategies. 


Implement the changes. The program can then implement one or more program changes that might 
help improve retention in parent education programs. The program could assess a single preferred 
strategy or assess two or three alternatives. 


Look at program data again to see if families’ program experiences are closer to expectations 
in the logic model. Program leaders can then check program data to see if the program changes helped 
re-engage parents sufficiently. If two or three alternative strategies were tested experimentally, the data 
could show whether one of the strategies was more effective than the others. 


Repeat the process as needed. If program changes do not increase parent engagement sufficiently, 
continue to brainstorm new strategies or refine the initial ideas, implement changes, and look at program 
data until families’ program experiences align with expectations in the logic model. 


Second, the data that programs obtain by examining the services received and participants’ feedback can provide 
insights on the design of the overall program (Exhibit 3). Program leaders and funders can regularly ask 
themselves whether the logic model really shows what they are trying to do—and revise it as needed. 


EXAMINE PARENT AND CHILD OUTCOMES USING THE LOGIC MODEL, AND ADJ UST THE 
SERVICES 


Once the program is running as expected, program staff and researchers can assess whether the parent and child 
outcomes are improving as anticipated. The outcome measures should be captured at the start of program services 
as a baseline and be checked against this baseline periodically. We suggest checking at least yearly—and more 
often if possible. More frequent measurement could help the assessment keep up with the rapid development of 
young children. 
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Exhibit 3. Practical examples of how stakeholders can compare data to the logic model 
and adjust the logic model 


Parent services examples: 

«* Ask a question based on the logic model. Program leaders may ask whether parents are receiving key 
employment services identified in the program’s logic model: one of the program’s three on-site job 
training activities and job search assistance. 


“+ Look at administrative data to answer the question. Program data might reveal that many parents 
are taking part in more than one on-site job training activity but they are not receiving job search services 
from a partner organization. 


“+ Ask parents and/ or staff why parents’ program experiences are not as expected. Program 
leaders and staff can ask parents why they are not participating in job search services and may learn that 
the services are difficult to access because of the location and times offered. 


«+ Consider strategies to bring program experiences in line with the logic model. Program staff or 
parents might have suggestions of changes in the schedule or location of job search services, but program 
leaders might decide that these changes are not possible because a partner is offering the services and is 
not willing to make changes. 


*“* Update the logic model. Leaders would then update the logic model to remove job search services from 
the logic model and add in job training services. They should then consider the possible effects of these 
service changes on shorter-term and longer-term outcomes. For example, the actual services received 
might lead to less short-term employment but to more long-term employment at higher-skilled jobs with 
better wages. 


Child services examples: 

«* Ask a question based on the logic model. Program leaders may ask whether children are attending 
the child development program for the two years they are eligible to attend, as reflected in the logic 
model. 


“+ Look at administrative data to answer the question. The program data might show that children are 
receiving child development services for one year rather than for the expected two years. 


“+ Ask parents and/ or staff why parents’ program experiences are not as expected. Program 
leaders and staff can ask parents why children participate for only one year and may learn that families 
move frequently. 


** Consider strategies to bring program experiences in line with the logic model. Program leaders 
and staff may not know how to engage families for longer because families are moving out of the area, 
and the early childhood education program’s wait list is long. 


“* Update the logic model. Program leaders might decide to update the logic model to show children 
receiving high quality education services for one year—and to modify their expectations for shorter- and 
longer-term child outcomes accordingly. 


When program leaders and researchers review short-term participation and outcome data, they should consider 
what changes can reasonably happen in a short time frame, and the possibility that outcomes could move in an 
unexpected direction during program participation. For example, the program might offer employment training 
services with the goal of seeing parents obtain higher-skilled jobs and higher wages. In the short term, however, 
while parents are enrolled in employment training, their rates of employment and their earnings might be lower 
than they were at baseline. 
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After reviewing data on outcomes for parents and children, program leaders and staff might conclude that the 
changes in outcomes are in line with the services that are offered, but they were hoping that the program would 
yield larger improvements in outcomes. Greater improvements in outcomes might require strengthening the 
quality, intensity, or duration of services. Research partners can help take stock of the services being offered, 
including the core features of the program’s service approach and its ties to evidence-based practices, and changes 
the program has made to fit the local context. They should also consult with program staff, participants, 
stakeholders, and communities of practice.? Researchers can also pinpoint what might be unique or novel about 
the services and whether they are being delivered with the quality or intensity needed to produce the desired 
outcomes. In identifying program adjustments, researchers could help program leaders consider strategies that are 
supported by evidence. 


Once program leaders and staff have identified feasible changes to the quality, intensity, or duration of services, 
they can implement them. If program leaders build evaluation into this process, they can assess whether the 
program changes are improving outcomes for parents or children. Evaluating whether the program change made a 
difference may be an important step if the program changes will increase service costs. Rapid-cycle evaluation is 
an approach that includes systematically implementing a program change and then examining outcomes to 
determine whether the program change made a difference (Exhibit 4). The research team can use rapid-cycle 
evaluation to assess whether program changes affect outcomes. The process might require more than one attempt 
to modify services before program leaders and staff reach a solution that improves outcomes. 


Exhibit 4. Practical examples of how stakeholders can use program data to assess the 
effects of changes in service approaches 


Outcomes that program leaders want to improve 

A majority of children who participated in the early education program are assessed at kindergarten entry as 
“not school ready.” Program leaders want to improve children’s language and social-emotional development 
by kindergarten. 


Practical examples of how to assess changes in service approach with program data 


Consider strategies to increase the intensity or quality of services. Program leaders and staff can 
brainstorm ways to strengthen early education services. Researchers can help by suggesting evidence- 
based strategies. 


Identify feasible strategies to implement. Two strategies program leaders and staff identify are to 
extend the half-day program to a full day and to provide professional development coaching for staff to 
improve teaching practices that support language development and social-emotional development. 


Implement the changes strategically so they can be tested. The program has 10 early education 
classrooms in five centers that offer half-day early education programs with extended-day care. The 
program leaders decide to pilot the full-day and staff development changes strategically so they can 
assess the impacts on children before deciding whether to implement the changes in all classrooms. They 
decide to implement full-day programs in 5 classrooms and professional development coaching in 4 
classrooms, with an overlap of both full-day programming and coaching in 2 classrooms. Three classrooms 
will continue without any of the changes. Children enrolling in the program will be randomly assigned to 
classrooms. 


Look at program data again to assess implementation and to see if families’ program 
experiences are in line with expectations. Program leaders can then check program data to see if 
teachers are receiving the expected level of professional development coaching and children are attending 
early education programs as expected. 

(continued on next page) 
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(continued from previous page) 

“* Examine outcome data to assess the impacts of the two strategies. The kindergarten assessment 
data can show whether one of the strategies was more effective than the other, or if the combined 
strategy was more effective than either strategy alone. 


*“* Repeat the process as needed. |f program changes do not demonstrate that the program changes 
improved children’s outcomes sufficiently, program leaders and staff could continue to brainstorm new 
strategies or refine the initial ideas, implement changes, and examine outcomes until they align with 
expectations in the logic model. 


USING RESEARCH PARTNERS AND PROGRAM PARTNERS TO 
SPUR PROGRAM DEVELOPMENT 


Programs that use data-informed program improvement may find that the process strengthens services and 
improves program leaders’ ability to make strategic program changes. However, the process may also require a 
sustained focus on cycles of data analysis and program improvement that can be difficult for program leaders to 
sustain when other pressing program demands arise. Partnerships between programs and external researchers can 
ensure that data systems development, analysis, and learning move forward amid other administrative demands on 
program leaders’ time. With an outside—and ideally, an objective—eye, researchers can help program leaders 
and staff assess program-specific findings in the context of other research on program development. For example, 
they can provide a broad perspective on refining program activities; finding evidence-based service approaches; 
and identifying ways to measure outcomes for adults, children, and families. 


In a scan of programs that coordinate services for parents and children (Sama-Miller and Baumgartner 2017), we 
found that most such programs did not have external research and evaluation partners, nor did they have in-house 
staff with data analysis capacity. Three programs that had in-house capacity used data to inform their program 
development (Exhibit 5). The two programs that had external research partners identified those partners early in 
implementation and infused research and evaluation into the process of developing the program. Partnerships such 
as these can help programs build their capacity to improve services. 


Partnerships with organizations that support peer networking can provide a community of practice that provides 
an opportunity for practitioners and program leaders to share ideas and work together to learn new ways to engage 
families, coordinate services, and address other issues. For example, the Strengthening Working Families 
Initiative of DOL supports peer sharing in various ways. They may participate in moderated discussion where 
experts weigh in on questions grantees pose about challenges. Grantees also may join facilitated conference calls 
that provide a forum for sharing challenges and solutions on specific topics, such as recruiting participants or 
connecting to child care resources. Exhibits 6 and 7 provide examples of such communities supported by federal 
grant programs and private organizations. Researchers who take part in these networks can support the learning 
community by sharing research-based best practices and promising innovations. Practitioners and program 
leaders, in turn, can share their knowledge about program operations and the types of changes that are feasible. 
They can also discuss shorter-term outcomes for parents and their children that they have observed, which could 
point to new ways to think about program effects, improving research and evaluation. 
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Exhibit 5. Research and evaluation capacity of three programs that coordinate services 
for families and children 


In-house data capacity. Next Generation Kids (http://jobs.utah.gov/edo/intergenerational), run by the Utah 
Department of Workforce Services, is an intensive case management program for families in the Temporary 
Assistance for Needy Families (TANF) program. It targets TANF families headed by adults who themselves 
received welfare benefits as children. To begin this pilot project, program leaders used longitudinal 
administrative data to identify the second-generation welfare beneficiaries who were eligible for the program. 


External research partners. College Access and Success (http://www. edalliance.org/adults) is a program 
that pairs Head Start/Early Head Start services for children with English as a second language (ESL) classes 
and help accessing postsecondary education for their parents. The program also partners with local university 
researchers to create program logic models, measure services and outcomes, use data to strengthen program 
implementation and assess outcomes, and conduct a formative evaluation. With foundation funding, College 
Access and Success is receiving technical assistance to help it use data more effectively. The program recently 
used participant feedback gathered by its research partner to improve access to career services. 


CareerAdvance® (https://captulsa.org/families/family-advancement/careeradvance/), a program that offers 


employment training to parents and Head Start services for children, has a similar partnership with 
researchers and has used data to inform the development of educational services for parents and their 
children. Staff and researchers worked to design services with aligned topics (for example, budgeting for 
parents coupled with early mathematics for children) to promote parent-child communication about learning. 


Both CareerAdvance® and College Access and Success, with their research partners, had impact evaluations 
under way in early 2017. 


These programs are among four selected for site visits and a fifth that the research team has worked with 
closely. The five programs informed our assessment of the evaluability of the range of programs that provide 
integrated services to adults and children and the features, contexts, and challenges of such programs. We 
selected programs that: (1) publicly reported a quality indicator for child services—such as a state education 
agency rating or National Association for the Education of Young Children certification—or provided EHS, Head 
Start, or an evidence-based home visiting program; (2) offered a range of adult education and workforce 
development programming; (3) operated with some federal funds or had state involvement in administering 
services; (4) offered a diverse array of services, and served a variety of target populations and locations; and 
(5) were not participating in other research requiring site visits at the same time. 


In some cases, programs may be able to analyze data in house. With dedicated staff, it’s possible to answer 
questions about service levels and quality, participant engagement, and outcomes by examining available data and 
strategically supplementing and improving these data. This information can help shape efforts to improve service 
quality, family engagement, service coordination, and other aspects of the program. 
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Exhibit 6. Examples of communities of practice organized by federal agencies 


Rural IMPACT Demonstration Sites, HHS, in collaboration with other federal partners. Ten rural and tribal 
communities receive technical assistance to develop comprehensive approaches to increasing parents’ 
employment and education and improving the well-being of children. Technical assistance includes peer 
learning networks that support web meetings and online sharing of documents to facilitate sharing information 
within and across sites (ASPE 2016). 


Strengthening Working Families I nitiative, U.S. Department of Labor. This grant program is helping 
communities strengthen the links between workforce development centers and affordable, quality early care 
and education to reduce barriers to employment. Grantees can participate in web meetings to facilitate 
discussion about challenges and solutions. 


Systems to Family Stability National Policy Academy, HHS, ACF’s Office of Family Assistance. This 18- 
month initiative provided technical assistance and support including peer sharing and learning to eight state 
and local TANF agencies that were implementing innovative redesigns of their TANF programs. One goal of the 
academy was to help programs approach case management in a way that took the whole family into account. 


Exhibit 7. Examples of communities of practice organized by private organizations 


Action Learning Network, Annie E. Casey Foundation. This year-long opportunity enabled programs in 
five communities to share best practices and provide support in program development. Participants in the 
network took part in the Family Economic Success—Early Childhood Initiative evaluation. 


Ascend Network, Aspen Institute. This network shares ideas about state policies to support the 
development of more coordinated services for parents and their children. It disseminates research on 
promising strategies, aims to influence policy decisions, and develops practice leaders in early childhood 
through postsecondary education and in health and human services. The network includes 180 partner 
organizations—researchers, policymakers, and program leaders interested in innovative approaches. 


Parents and Children Thriving Together: Two-Generation State Policy Network, Center for Law and 
Social Policy and National Governor's Association. Five states are taking part in a network while developing 
two-generation strategies for systems change. 


DESCRIBING PROGRAM SERVICES AND OUTCOMES 


Programs that are delivering strong services can inform the field, including other programs and policymakers, 
through descriptive research. Researchers can work with these programs to describe the quality and intensity 
(including the range) of services provided and the outcomes for parents and their children to spread the word 
about promising approaches. 


A descriptive study might focus on several research and evaluation questions that could provide important 
insights on promising programs in this nascent field: 


@ What is the quality of the child development services offered by the program? How does the quality compare 
to other early care and education programs in the community? To statewide standards of quality (for 
example, in the quality rating and improvement system)? To national standards of quality (for example, Head 
Start performance standards or quality standards from the National Association for the Education of Young 
Children)? To other programs that jointly serve parents and their children? 
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@ What is the intensity of child development services offered by the program? How does the intensity compare 
to other early care and education programs in the community? To other programs that serve parents and their 
children? 


@ What is the range and intensity of education and employment-related services the program offers for parents? 
Do the services meet industry standards for the intended types of employment? How do the content and 
quality of these services compare to other employment and education programs in the community and state?4 


@ How do child development and adult services complement one another in terms of content, delivery, 
schedules, and location? 


e@ What are the outcomes for children and parents one year, two years, and three years after enrollment? 


e@ What parent characteristics are associated with taking part in education services (such as English as a second 
language, adult basic education and General Educational Development credential [GED] preparation, job 
certification programs, and community college)? Parent characteristics could include education level; age; 
job experience; wage levels; and motivation to use employment, education, and children’s services. What 
parent characteristics are associated with participating in job training? With participation for a longer period? 
What are the patterns of participation in adult services? 


e@ What parent characteristics (age, education level, job experience, or motivation) and child characteristics 
(age, developmental status, or whether first born) are associated with longer participation in child 
development services? 


A study could focus on these questions for several programs that each take a different approach to combining 
services for parents and their children. Such a study could benefit the field by: 


e Informing program leaders about alternative program designs 


e Contributing more broadly to the knowledge base on the strengths and challenges of alternative approaches 


The study would be best able to meet these goals if the programs included were diverse, including different 
program backgrounds, types of populations served, and services provided. In addition, the programs included in 
the study should have more fully implemented program models. Such programs would have previously conducted 
some work to establish their program logic models, assessed how the services they provide line up with the logic 
model, and started to align their services with the expectations in the model. 


To describe parent and child characteristics, the types, quality, and intensity of services used, and outcomes for 
parents and children across programs, the study should use measures that are as consistent as possible. To ensure 
measures are consistent, it might require the use of data beyond that available in administrative data. 


The study could be a coordinated effort among researchers that are working with each program or an effort by a 
team of external researchers working with several programs. Data could come from the programs’ administrative 
data systems if the programs collect similar information on service quality and intensity, the initial characteristics 
of children and adults, and outcomes. Establishing a set of common “core measures” of key service and 
participant characteristics would help paint a consistent picture across the programs. Data on parents and children 
after they leave the program could come from other data systems, such as education or employment agency data 
(as noted earlier in this brief) and could provide consistent measures across programs. To fill in gaps where data 
are not consistent across programs, researchers could conduct additional data collection. 
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CONSIDERING WHETHER A PROGRAM IS READY TO MEASURE IMPACTS 


Measuring the impacts of programs that offer coordinated parent and child services is critical, given that few 
programs operating today have taken part in impact studies. Our literature review for this project showed very 
little research on the overall effectiveness of current programs, largely because most of these programs are still 
developing their service models and are not yet ready for an impact evaluation (for more information on the 
literature review and characteristics of current programs, please see the final report for this project, Sama-Miller 
et al. 2017). 


An impact evaluation can seem critically important to funders, who may be anxious to measure the value of their 
investments in these programs. It can assess whether the program is making a difference for parents and their 
children. For example, an impact evaluation can assess whether the program is changing outcomes in the shorter- 
term, such as parents attaining certifications or improving children’s early literacy development. It can also assess 
whether program impacts extend for a longer period, changing the trajectories of family economic well-being and 
children’s development over a decade or more. Many important questions about programs that offer coordinated 
parent and child services can be addressed by an impact study (Exhibit 8). 


Exhibit 8. Questions that can be addressed by an impact evaluation 


What are the impacts of a program offering coordinated parent and child services compared to other 
services in the community? How does the program affect parents’ employment, employment-related skills 
and education, and family economic security? How does the program affect parenting and parent-child 
relationships? How does the program affect children’s language development, early literacy skills and 
reading achievement, mathematics achievement, and social-emotional development? 


Is there a greater impact of coordinating services for parents and their children compared with offering 
services to parents and children separately? Put differently, does a program offering coordinated parent 
and child services have greater impacts on children than a program focused on just the children? Does a 
program offering coordinated parent and child services have greater impacts on parents than a program 
focused on just the parents? 


What are the critical levels of quality and intensity of services to parents and to their children that are 
necessary for programs to have an impact on parents’ employment, family economic outcomes, and 
children’s development? 


While the questions that an impact evaluation can address are important, funders and program leaders should not 
rush into impact evaluation before the program is ready. Long-standing evaluation theory stresses the importance 
of conducting impact evaluations with programs that are well-implemented and that have shown that participants’ 
outcomes are trending in the expected direction (Wholey 2010). Because many programs that coordinate services 
are still getting off the ground, while others are more fully implemented, judging a program’s readiness for an 
impact evaluation requires careful consideration. 


Here, we discuss the steps for assessing program readiness for an impact evaluation and the sources of 
information used in such an assessment. For most programs, a three-step “evaluability assessment” can show 
whether a program is ready for an impact evaluation (Exhibit 9). This assessment examines (1) the program’s 
level of implementation and (2) the strength of its theory of change. If the program has been working with 
researchers on data-informed improvements, the information should be readily available and the findings are 
likely to indicate readiness for impact evaluation. An evaluability assessment also examines (3) other factors 
related to evaluation capacity, which we discuss in detail below. Program stakeholders should be closely involved 
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with this step of the assessment, given their front-line perspective and intimate knowledge of the program. Staff 
buy-in to an impact evaluation is also vital to its success. 


Step 1: Assess the implementation status. In Step 1, evaluators can assess how well the program is put into 
practice (Metz et al. 2015; Metz and Albers 2014; Metz 2016). A program that is not implemented well is unlikely 
to yield positive outcomes, regardless of the positive intentions of leaders and staff. 


To assess implementation status, evaluators would examine various organizational factors, such as how a program 
hires, trains, and supports staff; what, how, and why program data are collected; how program decisions are made; 
how committed staff are to the program; and the program’s external partnerships and supports. Evaluators would 
also examine participant responsiveness, such as the number of enrollees and the amount of services they receive. 
This level of information — about exactly how the program operates, its staffing levels, staff training and support, 
and the actual level of services received — will also need to be documented during an impact evaluation. These are 
critical program “ingredients” needed to replicate the program if an evaluation finds that the program had 
favorable effects on parents and their children. 


Step 2: Assess the strength of the program’s logic model. A clear logic model is vital for programs that hope to 
show positive impacts. This important tool can help researchers and program leaders gauge the strength of the 
program’s vision for service quality and the expectations for parent and child outcomes. As discussed earlier, 
program leaders and staff, working with researchers, can develop this model and then use program data to assess 
how well the types, intensity, and quality of the actual services line up with their expectations. Researchers and 
staff can also use the logic model and outcome data to assess whether the outcomes for parents and children are 
trending in the right direction. Building on the work that programs may have done in partnership with researchers, 
the logic model and analyses of program data supporting continual improvement would be helpful in making the 
case that the program is ready for impact evaluation. 


Step 3: Consider the maximum possible rigor of the evaluation design. If a program passes the first two steps, 
an evaluator should determine the most rigorous research design that the program can support. Randomized 
controlled trials (RCTs) are the most rigorous design option. A program may not have the enrollment level and 
staff capacities needed for an RCT (examples of such capacities include enrolling twice as many families in the 
evaluation as will receive services and collecting additional data at enrollment to assist with maintaining contact 
with families during the follow-up period). If enrollment levels and staff capacities are not sufficient to support an 
RCT, the next-best solution may be a quasi-experimental design (QED), which requires finding a very similar 
group of families to compare to program participants. QEDs do not produce the strongest causal evidence about 
impacts but may be the best choice for emerging program models. 
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Exhibit 9. Three steps to assess a program’s readiness for an impact evaluation 


‘Step 1: Assess implementation status 


> Competency drivers: > Fidelity of implementation: 
Staffing and training Staff knowledge and buy-in, 
> Organization drivers: quality assurance, staff 
support 


Leadership, community 

partnerships, data systems » Participant responsiveness: 
Enrollment and 
participation 


Step 2: Assess the strength of the program’s logic model 


> Are the types of services for parents and children designed to achieve desired outcomes? 


> Are the quality and intensity of services sufficient to achieve the desired outcomes? 


> Is there research to support anticipated pathways from service components to outcomes? 


> Does the logic model indicate sufficient resources and inputs and a realistic time frame for 
seeing outcomes? 


Step 3: Consider maximum possible rigor of evaluation design 

> Target population: number of potentially eligible families in the community exceeds program capacity and current and potential referra 
sources can expand the number of program applicants 

> Program capacity: number of families who can be served, ability to increase capacity if needed 


> Evaluation capacity: ability to enroll and randomly assign families from different sources, maintain family random assignment over time, 
interest in evaluation among program leadership and staff, partner and community support for evaluation 


> Evaluation fidelity: ability to deliver a positive message about the evaluation and adhere to random assignment outcomes 
> Data capacity: willingness and ability to assist with data collection at enrollment or baseline and at other points later in the evaluation 
> Community context: differentiation from other services in the community 


Note: Implementation criteria are derived from Metz and Bartley (2012) and Metz et al. (2015); other criteria 
are derived from Davies (2013), Dunn (2008), and Wholey (2010). 


Evaluators should consider several factors before selecting a design for an impact study: 


e The size of the potentially eligible population in the community and the current referral sources for the 
program (to assess the potential to expand the number of applicants through broader outreach and referrals, 
to meet sample size requirements, and to form a control group) 


e Program capacity to provide services for the number of participants who need to be served (researchers can 
estimate the number that need to be served in order to measure any impact of the program that may exist) 


e@ Leadership, staff, and community support for the evaluation 


e Staff interest and ability to conduct evaluation activities, including enrolling families that might apply 
through either child services or adult services, providing clear and positive messages to families about the 
evaluation, and maintaining random assignments over time 


e A strong contrast between the program’s services and the generally available services in the community 


Data sources to inform the three assessment steps. Some of the information needed to assess whether a 
program is ready for an impact evaluation could come from publicly-available data sources, including program 
websites, grant proposals, and reports. These sources might provide information on the number of families served 
each year, eligibility criteria for participation, funding sources, implementation challenges, and outcomes for 
parents and children. However, these sources might not always include enough detail or provide the most recent 
information about the program. 
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To obtain a more complete assessment of the program’s readiness for evaluation, two other approaches might be 
needed: 


e@ Telephone interviews with program leaders and researchers and additional document reviews. 
Telephone interviews with program leaders can help clarify the types of services provided to parents and 
children and the number of and qualifications of staff. Telephone interviews with research partners can 
provide details on the types and quality of services received, the levels of attendance by parents and children 
in educational activities, and short- and medium-term outcomes. Researchers might also have access to 
program data that enables them to provide customized data summaries that can address questions such as 
whether the parents and children served by the program are usually from the same families or from different 
families. 


@ Site visits. Site visits can offer stakeholders a better understanding of the program staff’s buy-in and interest 
in evaluation, the quality and intensity of services, and the existing and potential capacity of the program to 
support an evaluation. Moreover, a visit can foster program staff’s interest in taking part in an impact 
evaluation. It can allow the evaluator or funder to explain the research benefits and process to program staff 
and to address their concerns about participating in a random assignment evaluation. 


If this assessment indicates that the program is ready for an impact study, researchers should select an evaluation 
design that can answer the research questions of greatest interest. 


CONCLUSIONS 


Research and evaluation can be beneficial to programs that combine services for parents and children. Such 
research can: 


e Help program leaders create service approaches aligned with their vision for the program 
e Inform the field about promising strategies 


e Eventually, address questions about the impacts of these programs on parents’ economic security and 
children’s well-being 


Importantly, the type of research and evaluation should be appropriate for the program’s stage of development. 
Most current programs that target parents’ economic security and children’s well-being are in the developmental 
stages. Thus, the research and evaluation activities most useful for them are descriptive approaches that support 
stronger program implementation. This type of research and evaluation can help programs develop logic models; 
strengthen their data systems; and use the data to assess program activities, outcomes, and costs. This assessment, 
in turn, can pinpoint priority areas for program development. 


Program leaders can engage in continual improvement as well as rapid-cycle testing of the improvements to see 
how they are working. Partnering with researchers, other programs, and communities of practice can also help 
program leaders develop services that line up with their goals for the program. Programs that are delivering strong 
services can participate in descriptive research that can inform the field, including other programs and 
policymakers, about service strategies that engage families and show promising outcomes for parents and 
children. 
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ENDNOTES 


1 http://www.jbassoc.com/reports-publications/dohve) and the Early 
Head Start Research and Evaluation measures compendium 
(Kopack Klein et al. 2017) 
https://www.acf.hhs.gov/sites/default/files/opre/ehsrep_measures 
final_508.pdf. 

? The logic model can also include longer-term outcomes for parents and 


children, but program administrative data is unlikely to be the most 
cost-effective and accurate way to measure these outcomes. 


3 A community of practice is a real or virtual space in which program 
leaders can share ideas and experiences and network with peers 
about solutions. 


4 The quality of adult education and job-related services is of interest, 
but there are no national or state standards for quality to use as 
benchmarks. A study can address questions of quality by 
comparing the services to what is required for jobs in the target 
industry and by obtaining feedback from participants about their 
experiences. 
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